1. Field of the Invention
The present invention relates generally to computer software design and quality control, and, in particular, the present invention relates to software certification systems and methods.
2. Background of the Invention
The concept of building “operational profiles” has long been used in major industries, such as aviation and telecommunications, to certify that a particular product or system will operate reliably and accurately for its intended purpose. In those industries, expensive data gathering techniques and test procedures can be justified due to the critical nature of the systems. For example, the failure of some aircraft component or subsystem could result in loss of life or hundreds of millions of dollars in lost revenue or equipment repairs. Similarly, the failure of a telecommunications node could result in service loss for millions of customers. Accordingly, such high profile, high risk industries can afford, and indeed in many cases are legally required, to perform sufficient testing to provide some level of quality assurance. Other industries have also been able to gather sufficient testing data in cases where the cost-profit margin is very large, or the systems are predictable enough to contain the testing expenses.
Another reason some industries have been successful in building accurate operational profiles is the wealth of historical data available. For example, the telecommunications industry has been able to collect profiles from thousands of “user years” over decades of calendar time. Similarly, after nearly 100 years of building aircraft, there are few operational anomalies that are not known to aircraft manufacturers. Historical data is important in providing a baseline against which testing and certification can be judged.
When sufficient testing is conducted the products may be “certified” either by the vendor, the government or some independent third party entity. Such certification may state that the products will perform to a certain level of reliability and accuracy (also referred to herein as “correctness”). The certification may even be accompanied by a warranty to insure users against product failures.
Unlike the industries described above, a typical vendor of commercial off-the-shelf (COTS) software cannot currently certify its software products without assuming enormous risks. This has been true for many reasons, but the primary cause has been the inability to predict every case that a software product will encounter in actual use by the population at large. In the software industry, an operational profile would be defined as a set of input events and operating environment that the software will be exposed to during execution, along with the probability some result will occur. This definition has worked fairly well for embedded software, i.e., software developed for limited purposes that is customized for specific hardware environments with fixed memory, fixed disk space, fixed processor speeds, and the like.
With COTS products, software testing and certification has not been practical due to the number of variables involved. COTS software is typically designed according to the operating platform upon which the software runs. That is, most COTS products are designed to be hardware platform independent. However, the reality is that the total environment, i.e., hardware, operating system, other applications concurrently running on a system, and the user's interaction with the system, all affect the operational profile of a COTS product. In order to confidently certify that a given COTS product is accurate and reliable, a means for gathering sufficient testing data for the wide variety of operational environments is needed. Additionally, certification would require a system and method for tracking and analyzing the collected data to formulate a reasonable probability estimate of the software's reliability and/or accuracy.
Another reason that the software industry has not been successful in providing certification for COTS products is the lack of historical data. The software industry is still in its infancy in comparison with the aircraft and telecommunications industries. The resulting dearth of historical data leaves each software vendor on its own in any efforts to establish a certification program or process.
Without such a certification process, COTS consumers, both business consumers and individual consumers, must rely on the reputation of the software's developer or vendor, marketing campaigns, anecdotal evidence from colleagues, and published software reviews in order to decide which software applications to buy. Even published reviews rarely deal with the quality of the software, nor can they since they do not have adequate time or resources to fully test the software. As a result, consumers have no independent, third-party appraisal of the quality of software, on which they may be heavily dependent, prior to buying or installing the software. As used herein, the terms software vendor, software publisher and software developer are used generally to mean the entity responsible for providing a some software product to consumers, either directly, or indirectly.
As described above, software vendors today can only test their software in a limited number of configurations, in a limited number of environments. Such limited configurations and operating environments provide software vendors with only limited input data for their testing processes. To maximize efficiency, software vendors generally develop and test code for a set of generic, “mythical,” users. This results in software products that are riskier for all users, because the products have not been tested according to how they will actually be used in the field.
In addition to testing software according to the mythical user, some COTS software vendors have attempted to provide quasi certification by certifying that their software development processes or personnel conform with standards set forth by various organizations. For example, the International Standards Organization has promulgated ISO9000 and the Software Engineering Institute has developed its Capability Maturity Model (SEI-CMM). In such approaches, software vendors usually take oaths concerning which development standards and processes were used, and auditors may “spot check” a vendor's project documentation to ensure that such documentation is consistent with the oaths taken. However, even when a software developer is truthful and follows a development cycle consistent with a given standard, high quality software is not guaranteed.
More recently, other software developers have attempted to create accurate operational profiles by monitoring application environments. One such example is a product called PureVision, which was offered by Pure Software, and released in 1995. PureVision allowed a software vendor to produce software that was able to monitor itself when running on a user's computer. Each copy of the software would send back a report to the vendor which included a list of the users at a given site who used the product, the software version number, system configuration, times when the software started and stopped executing, which program features were used, and the amount of memory used at exit. In addition, if the product failed, exit codes and a stack dump were added to the report.
Pure Software knew that users would be wary of vendors looking over their shoulders and thus included an option by which a user could inspect a report before it was sent back, as well as an option to not send a report back at all. It is speculated that PureVision did not survive because users were unwilling to provide or uncomfortable providing such detailed, non-technical information to outside groups. Moreover, the product itself could not provide software certification because it only gathered the data, but did not provide any analysis.
Another product previously used in the art is The Netscape Quality Feedback Agent, which began shipping with version 4.5 of Netscape Navigator (available from Netscape Software, Inc). The Netscape Quality Feedback Agent sends feedback to Netscape's application developers concerning how the product is performing. The agent is enabled by default and is activated when the software encounters some type of run-time problem. The agent collects relevant technical data and displays a form in which a user can type comments. Netscape intends to use this data to debug known problems and identify new ones as they arise. However, as with PureVision, the agent raises many privacy issues that leave consumers with little assurance that their personal information is protected. Moreover, the agent also relies on the willingness of users to actively provide comments, fill-out forms and actually submit the data, when an error is detected.
Another software testing model used in the prior art is the distribution of pre-release, or “beta,” copies of software to pre-qualified users in exchange for feedback concerning product stability, usability, and reliability. For example, the Microsoft Corporation has long employed beta-testing as a way to collect information on how their products perform in real world environments. Microsoft uses this information to decide when a product is ready for general release.
Finally, although not technically an operational profile system, the so-called “open source” model for software development has resulted in a greater degree of software reliability in some cases. The classic example is that of the Linux operating system. Linux is a Unix operating system project that is the product of hundreds of users, all of whom donated their time to write and test the system. Linux is considered to be the most reliable of all Unix operating systems. In fact, the success of Linux is often used as the argument for why an open source model should be used by other companies. Even so, the open source model does not certify software and no vendor or other entity has certified or warranted Linux as a highly reliable and accurate operating system software.
In each case described above, a fundamental flaw in the software testing and “certification” process is the strong reliance on the software vendor or auditors to certify that certain procedures have been followed, or that a sufficient amount of data has been gathered to provide a statistically sound analysis of a product's reliability. One way to overcome this flaw is to establish an independent “software certification laboratory” (SCL) that tests software and issues a “certificate” if a product meets some pre-defined criteria. The idea of an SCL is not new, but the formation of such an entity has not been successful in the past for practical reasons. A key reason why SCLs have not become widespread is the liability of a “certifier.” Such liability arises from the fact that when certified software fails in the field, the certifier bears some level of liability due to the certifier's representations as to the quality of the software.
To reduce an SCL's liability, accurate methods for making certification decisions must be employed. Unfortunately, even the best statistical analysis and testing techniques often fail to consider the actual stresses that software will experience when used in the field by real end-users. Such stresses may arise, e.g., due to user error, or unusual hardware or software configurations on which the software must run. Thus SCLs suffer from the problem of accurately determining how well-behaved a software product will be in the future. Such anticipated behavioral characteristics are a key piece of information needed by an SCL before it could offer a meaningful certificate of the software.